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Doel van het onderzoek was het verkrijgen van inzicht in de effecten van verschillende vormen 
van continue haptische informatie (CHI) op de prestaties van operators van onbemande platforms 
die zijn uitgerust met een camera-monitor systeem. In een experiment met de TNO-TM RPV 
simulator moesten proefpersonen een gesimuleerd bewegend (verstoord) platform zo goed mogelijk 
recht boven een bewegend doel houden. Tijdens het uitvoeren van deze besturingstaak werd hapti¬ 
sche informatie gegeven over de translates van het platform (en daarmee van de camera). 

In het experiment werden twee besturingsvariabelen (vier besturingscondities) between-subjects 
gemanipuleerd. Dit waren: de aan- of afwezigheid van CHI en van (geautomatiseerde) stabilisatie 
van het platform. Hiervoor werden vier keer veertien proefpersonen gebruikt. Drie variabelen 
werden within-subjects gevarieerd, i.e., visuele degradatie door vier verschillende update 
frequenties van het camerabeeld, motorisch taakbelasting door hoge vs. lage beweeglijkheid van 
het doel en visuele taakbelasting door aanwezigheid of afwezigheid van een extra visuele aandacht- 
taak. 

Proefpersonen zaten gedurende het experiment in een afgesloten ruimte met een beeldscherm voor 
zich. Hierop werd de visuele bewegingsinformatie gegeven. De aanvullende haptische informatie 
werd gegenereerd door de uitslagen van een zogenaamde actieve joystick. Hiermee moest door 
middel van "force control" tevens de camera bestuurd worden. 

Analyse van de stuurfouten toont aan dat CHI een kleinere volgfout bewerkstelligt evenals 
automatische stabilisatie van het platform. Deze effecten waren niet additief. In combinatie waren 
de effecten niet groter dan apart. Terwijl het effect van platformstabilisatie afham (maar significant 
bleef) naarmate de update rate hoger werd, bleken de effecten van update rate en CHI additief. 
Wanneer de motorische of visuele taakbelasting toenam, door een beweeglijker target of door een 
extra visuele taak, steeg tevens het effect van platformstabilisatie. De grootte van het CHI effect 
bleek onafhankelijk van taakbelasting. 

Belangrijkste conclusie: als CHI via het besturingsmiddel betrekking heeft op besturingsinformatie 
leidt dit tot een betere stuurprestatie; als het informatie betreft die minder direct relevant is voor 
besturing, bijvoorbeeld ter verbetering van het situationeel bewustzijn, dan interfereert het niet 
substantieel met de stuurprestatie. 
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SUMMARY 

The present study was conducted to gain insight in the effects of different forms of continu¬ 
ous haptic information (CHI) to operator performance of an unmanned platform equipped 
with a camera-monitor system. In a simulator experiment, subjects had to track a moving 
target keeping a simulated disturbed platform as well as possible straight above a moving 
target. While performing this tracking task, haptic information was provided concerning 
translations of the platform (and thereby of camera). 

Two independent control variables were manipulated between-subjects (four control 
conditions). These factors were: presence or absence of CHI and of (automated) stabilization 
of the platform. Each of the four groups consisted of 14 subjects. Three other factors were 
varied within-subjects, i.e., image degradation by four different image update rates, motor 
taskload through the amount of target motion, and visual taskload by the presence or absence 
of an extra visual attention-task. 

During the experiment subjects were seated in front of a monitor in a closed room. This 
screen was used for visual information presentation. The supplementary haptic information 
was generated by movements of a so-called active joystick which at the same time was used 
for steering the camera by “force control”. 

Analysis of the tracking errors shows that CHI as well as automated platform stabilization 
substantially reduced tracking error. These effects were not additive as the combined effects 
of CHI and stabilization were not larger than the separate effects. Whereas the effect of 
stabilization diminished (but remained significant) with increasing update rate, the effects of 
CHI and update rate were additive. With increasing motor or visual taskload, as a conse¬ 
quence of an extra visual task or increased target motion, the stabilization effect increased. 
The magnitude of the CHI effect appeared independent of both forms of taskload. 

Major conclusion: CHI at the control device improves tracking performance when it 
involves information that can be used directly for vehicle control; when it provides informa¬ 
tion that is less relevant for control, tracking performance is not substantially degraded. 
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Rap. nr. TM-96-B012 


TNO Technische Menskunde 
Soesterberg 


Continue haptische informatie en platformstabilisatie in een stuurtaak 

J.E. Korteling en M. van Emmerik 

SAMENVATTING 

Om inzicht te krijgen in de effecten van verschillende vormen van continue haptische 
informatie (CHI) op de prestaties van operators van onbemande platforms moesten proefper- 
sonen een gesimuleerd bewegend (verstoord) platform zo goed mogelijk recht boven een 
bewegend doel houden. Tijdens het uitvoeren van deze besturingstaak werd haptische 
informatie gegeven over de translates van het platform (en daarmee van de camera). 

In het experiment werden twee besturingsvariabelen (vier besturingscondities) between- 
subjects gemanipuleerd. Dit waren: de aan- of afwezigheid van CHI en van (geautomati- 
seerde) stabilisatie van het platform. Hiervoor werden vier keer veertien proefpersonen 
gebruikt. Drie variabelen werden within-subjects gevarieerd, i.e., visuele degradatie door 
vier verschillende update frequenties van het camerabeeld, motorische taakbelasting door 
hoge vs. lage beweeglijkheid van het doel en visuele taakbelasting door aanwezigheid of 
afwezigheid van een extra visuele aandacht-taak. 

Proefpersonen zaten gedurende het experiment in een afgesloten ruimte met een beeldscherm 
voor zich. Hierop werd de visuele bewegingsinformatie gegeven. De aanvullende haptische 
informatie werd gegenereerd door de uitslagen van een zogenaamde actieve joystick. 
Hiermee moest door middel van “force control” tevens de camera bestuurd worden. 

Analyse van de stuurfouten toont aan dat CHI een kleinere volgfout bewerkstelligt evenals 
automatische stabilisatie van het platform. Deze effecten waren niet additief. In combinatie 
waren de effecten niet groter dan apart. Terwijl het effect van platformstabilisatie afnam 
(maar significant bleef) naarmate de update rate hoger werd, bleken de effecten van update 
rate en CHI additief. Wanneer de motorische of visuele taakbelasting toenam, door een 
beweeglijker target of door een extra visuele taak, steeg tevens het effect van platform¬ 
stabilisatie. De grootte van het CHI effect bleek onafhankelijk van taakbelasting. 

Belangrijkste conclusie: als CHI via het besturingsmiddel betrekking heeft op besturings- 
informatie leidt dit tot een betere stuurprestatie; als het informatie betreft die minder direct 
relevant is voor besturing, bijvoorbeeld ter verbetering van het situationeel bewustzijn, dan 
interfereert het niet substantieel met de stuurprestatie. 
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1 INTRODUCTION 

Over the past decades the concept of teleoperation has evolved from fiction to reality. 
Developments in computer technology (miniature electronics) and satellite navigation 
systems like GPS (Odum, 1995) combined with strong and light materials from space 
technology made it possible to construct for instance small unmanned aerial vehicles 
(UAV’s) equipped to perform all kinds of missions in environments potentially harmful for 
humans. But instead of entering a Utopia in which subservient machines willingly act on our 
wishes at the push of a button, we see ourselves faced with new jobs, new questions and 
above all new problems. These problems primarily concern a lack of coherent sensory 
information specifying the motions of the platform-camera system in relation to the environ¬ 
ment and other (moving) objects in it. 


1.1 Perceptual problems in platform control 

During unmanned platform control the operators are at a distant location and thus lack direct 
sensory information. To perform his basic tasks (vehicle guidance and information acquisi¬ 
tion) the operator depends on “substitute senses” provided by technology. In so far as 
technology is not (yet) capable to stimulate all of our senses accurately, this means practi¬ 
cally that only visual information as supplied by a camera-monitor system is available to the 
operator. Consider on the other hand the sensory information available to the pilot of a 
manned aircraft: "... [T]he wide visual field provides the pilot an immediate indication of 
the slightest attitude change (visual proprioception). Small translational accelerations of the 
aircraft are felt by the vestibular organs. The aerodynamic forces on the control surfeces are 
felt by the pressure and stretch receptors of the hand and arm. Speed can be inferred from 
aerodynamic noise or by the sound of the engine.” (Korteling & Van der Borg, 1994, 
1997). The combination of senses provides the pilot with a fairly unambiguous picture of the 
world (Patrick, Sheridan, Massimino & Marcus, 1990). The platform operator thus lacks 
this abundance of proprioceptive information. This will make his task more difficult and 
likely more demanding. 

Mission-specific aspects provide an additional problem. A space-platform, for example, 
operates at such a distance from its control station that signal transmission suffers from 
delays (Stokes, Wickens & Kyte, 1990). Sometimes only limited bandwidths can be used 
because the signal has to be digitized and coded. This is common practice in the navy to 
decrease vulnerability to disturbances and to prevent enemy jamming (Korteling & Van der 
Borg, 1994, 1997). Operating and interpreting the camera image will be even more 
complicated as a consequence of restricted field of view and low resolution. Poor image 
quality may lead to difficulties in separating target motion and “self” motion. Furthermore, 
when the operator is unable to discriminate between movements of the platform and 
movements of the camera (as viewing direction and platform movements are controlled 
independently) this will lead to disorientation. Due to the lack of sensory information as well 
as to the problems just mentioned, the operator might experience serious problems with his 
situation awareness, i.e., the perception of orientation and position of the aircraft and/or the 
sensor in space and time together with an apprehension of the environment (threat, targets), 
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flight and system conditions (Van Erp, Korteling & Kappe, 1995). It is hardly necessary to 
point out that this will be obstructive to mission performance. In addition, it has been 
pointed out that automation may reduce situation awareness, which may be an important 
factor underlying accidents in complex man-machine systems (e.g., Endsley, 1995; Wickens, 
1992). 


1.2 Continuous haptic information 

In the early days of aviation, cables and push-rods directly linked the aircraft’s control to its 
control surfaces. The stiffness of the control column then arises partly from the aerodynamic 
forces on the rudder. When these forces are subject to change (e.g. as a result of accelera¬ 
tion) the stiffness of the control column will also change, by that providing feedback to the 
pilot (Van Paassen, 1994). 

For UAV control such a direct-link control system is not feasible. The operator’s commands 
can no longer be transmitted mechanically. Instead deflections of a joystick are translated 
into electrical signals that are fed into a flight control computer. The computer calculates the 
desired rudder change that is, in turn, passed on to the servo motors controlling rudder 
angle. Such a control system is called fly by wire. In these systems, the computer’s task is to 
stabilize the aircraft. The absence of direct mechanical links between control and control 
surface, however, allows for a considerable amount of manipulation of the joystick input. 
This gave rise to the development of active steering devices. An active joystick consists of a 
force sensor attached to a joystick. The joystick itself is moved by motors. The motors get 
their input from a computer that can, with the appropriate algorithm, be coupled to any 
variable of aircraft motion. In this way the operator can be informed about altitude changes, 
climb speed, direction changes of the aircraft etc. through the position of the stick (Korteling 
& Van Gent, 1994; Van Paassen, 1994). The success of an active stick will depend on a 
proper choice of the feedback variable and the manipulation of joystick input. 

With the availability of advanced computer systems an obvious next step in UAV-control 
would be to automatize the task completely. This is not without problems however. 
According to Wickens (1992) the concept of flexibility is of great importance. There seems 
to be a trade off between the range of capabilities and the degree of flexibility of a man- 
machine system (Korteling & Van der Borg, 1994, 1997). Automation requires that a 
mission is planned and programmed a priori. Tasks like controlling a UAV are sometimes 
difficult to plan in advance due to pursuit tracking aspects. They require a human operator 
to obtain the necessary level of flexibility. It seems then, the optimal solution is partial 
automation of platform-control. Although this will relieve the operator’s taskload, it places 
the operator partially outside the control loop which is considered a potential drawback of 
automation. Without precise knowledge of “what the system is doing, why it is doing it and 
what it will do next”, a loss of situation awareness may be the result (Wickens, 1992). 

Application of CHI may enhance situation awareness of remote operators. The potential 
benefit of haptic information has also been demonstrated in other fields such as car driving 
and aircraft control. In a study on car driving, continuous force feedback concerning the 
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lateral position of the car on the road was provided through an active steering wheel. In this 
experiment a reduction of both control error (deviation of lateral position) and control effort 
(driver’s steering wheel movements) was observed (Schumann, Lowenau & Naab, 1994). In 
an exploratory simulator study, haptic feedback through an active gas-pedal was found to be 
of help in maintaining a constant speed, thereby reducing taskload (Schumann, Godthelp & 
Hoekstra, 1992). 

An active stick was also found to be an aid in aircraft control (Laurman & Verhoeven, 
1993). In a simulator study, subjects had to approach a runway. Subjective taskload ratings 
(Cooper-Harper scale) and standard deviations from glide path and speed during this 
approach were used as an index of task performance and compared for different controls. 
The active stick informed subjects of either roll speed and heading angle velocity or roll 
speed and vertical disturbance speed. Although both subjective ratings and standard 
deviations indicated superior performance with the active controls compared to a passive 
stick, methodological limits of the experiment made it impossible to reliably differentiate 
between the specific feedback algorithms. 

Using auditory and force feedback, Massimino and Sheridan (1993, 1994) tested subjects 
with several peg-in-hole telemanipulation tasks. In these tasks, subjects had to put a peg into 
holes of different size using a remotely operated robot arm which could be controlled in 7 
degrees of freedom. The authors concluded that force feedback could compensate for 
conditions of degraded visual information. The effect of force feedback on task time at an 
image update rate of 3Hz was found to be of the same magnitude as an increase in update 
rate of the visual information from 3 to 30Hz without force feedback. For better visual 
conditions (i.e. higher update frequencies) the effect was less pronounced. 

Results in a study by Korteling and Van der Borg (1994, 1997) are in agreement with these 
findings. In their study they had subjects to operate a semi-autonomous interface for a 
simulated UAV. The system automatically compensated for translations of the platform 
relative to the earth and accompanied these system interventions by the appropriate joystick 
movements. As a result subjects received tactile (haptic) information of the compensatory 
interventions of the system. The magnitude of the active-interface effect was equal to the 
effect of an increase in update-frequency from 2 to 5Hz. 

A study concerning helicopter control also found a beneficial effect of haptic information 
(Korteling & Van Gent, 1994). The effect was limited, however, to altitude maintenance and 
appeared most pronounced in high taskload conditions. It is therefore suggested that haptic 
information might particularly be of help to compensatory control tasks in demanding 
situations (Kelley, 1968; Korteling & Van Gent, 1994). 


1.3 The present experiment 

The present study was undertaken to investigate the effects of continuous haptic information 
(CHI) in a tracking task. It was investigated if CHI could contribute to reducing operator 
taskload and would facilitate operator performance. It was also considered of interest to 
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evaluate the usefulness of CHI in conditions of high automation. Under these very circum¬ 
stances, information on the nature of the system’s actions may help to keep the operator in 
the control loop, i.e., enhancing his situation awareness (e.g. Korteling & Van der Borg, 
1994, 1997; Wickens, 1992). 

In the present experiment subjects had to track the target in a task situation that, in an 
abstract way, represented the tracking of a moving target from a moving platform with a 
camera-monitor system. Subjects used either an active joystick providing CHI or a passive 
joystick. These joystick types were combined with two levels of stabilization (either subjects 
had to stabilize the platform while tracking the moving object, or the stabilizing task was 
done automatically by the connected computer systems). This resulted in four task conditions 
(or stick modes). 

By manipulating the update frequency of the monitor image, perceptual degradation was 
varied. Motor load was controlled by varying target mobility. It was investigated in which 
way these factors affected task performance in interaction with CHI and automation. 
Furthermore, a secondary task was developed. This task forced subjects to divide visual 
attention, however, without causing structural interference (by visual scanning require¬ 
ments). Decrease in tracking task performance due to extra visual attention requirements 
(visual load) was expected to be counteracted by availability of CHI. 

Because platform control is complicated by a number of independent control variables, (e.g. 
camera angle, viewing direction and vehicle heading) in this study a simplified 2D paradigm 
of tracking, without stereoscopic- or perspective information was favoured over a more 
complex 3D paradigm. By assuming the camera to be aimed perpendicular to the earth no 
perspective or other depth cues were involved. With the camera angle fixed, platform 
control and camera control were completely dependent. It was expected that this simplifica¬ 
tion would allow for measurement of the effects of CHI on platform control with a minimum 
of interference from other (complicating) factors. 

One potential drawback of the 2D image is a reduced correspondence with the real world 
situation. This is however regarded of minor importance as the main objective was not to 
simulate a real operator room. The study was primarily focused on the possibility of using 
haptic information as a supplement to visual information, in an object-tracking including 
disturbance motions of the (own) platform. 

With regard to the results, clear effects of image update frequency, target mobility and 
visual attentional load were expected. More interesting, however, were the differences 
between the stick configurations and their interactions. It was expected that tracking error 
would become significantly smaller with the active stick configurations compared to their 
passive counterparts. In related research the benefits of haptic information were most 
pronounced under extreme and difficult circumstances. Therefore, it was expected that the 
aforementioned advantage would decrease in the configurations with automatic platform- 
stabilization. The stick-mode effects were expected to increase again with conditions of low 
image update rate, increased motor- and visual load. 
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Furthermore, if the results would indicate minimal negative effects of CHI with automatic 
platform stabilization, the idea of enhancing situation awareness by providing CHI in semi- 
automated control tasks will be a promising opportunity requiring further investigation. 


2 METHOD 
2.1 Instrumentation 

The experiment was conducted in the TNO-TM Teleoperation Research Simulator. This 
facility is especially designed for simulating teleoperation missions, such as with remotely 
piloted vehicles (Korteling & Van Breda, 1994). Detailed specifications are given in 
Appendix A. In the present experiment, the chair was placed at a marked spot to make sure 
subjects kept a roughly equivalent distance from the monitor. The subjects had a joystick to 
their right and a push-button to their left. Straight in front of them, at a distance of approxi¬ 
mately 80cm, a 19in RGB high-resolution monitor (Mitsubishi colour display monitor) was 
placed. The foot pedals of the mock-up were temporarily removed. 

Joystick control was achieved by two independent servomechanisms, which received their 
input from a force sensor. Maximum stick deflection range was +/— 30°. Only one-third of 
the total stick range could be used for control (simulation of stick suspension) because the 
re maining two-third had to be reserved for stick-motion information in the active-stick 
conditions. To obtain this one-third of total stick range a force of 2 N had to be applied to 
the stick. Maximal speed of compensation for disturbance error was 7.95cm/s. The relation 
between joystick force (N) and screen speed was linear. The joystick extended 9cm above its 
pivot. A 486 PC (IBM compatible) was used to compute the changes in target position 
caused by force input on the stick. This information was passed on to a Silicon Graphics 
computer (IRIS 4D) and was used to generate the monitor image. Another 486 PC (IBM 
compatible) was used for scenario generation and was responsible for data storage (10Hz 
sampling frequency). 


2.2 Subjects 

The experiment was conducted with four groups of 14 right-handed male subjects with high 
educational level (VWO/HBO or university). Mean age of the subjects was 22 years and 11 
months (range 17-30, SD 2.48). All subjects had normal or corrected-to-normal vision. 
Subjects were matched for “information processing speed” (Salthouse, 1985) as measured by 
the WAIS-subtest “Digit-Symbol Substitution” (Wechsler, 1981). The subjects were paid for 
their participation. 
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2.3 The experimental tasks 

Subjects were seated at a distance of approximately 80cm from the monitor. They had to 
track a target in a task situation that, in an abstract way, represented the tracking of a 
moving target from a moving platform with a camera-monitor system. The image on the 
screen represented an image of a flat background (environment) with one target), as seen by 
a camera fixed on a (moving) platform. Platform motions were parallel to the flat back¬ 
ground. The camera was aimed perpendicular to the environment. As a result of this 
viewing-angle the image represented a 2D picture. The target and the camera on the 
platform only translated in the x- and y- axis. There were no rotations. The background was 
created by a hundred plus signs, randomly distributed over a black screen. A white cross¬ 
hair (0,4°), representing the visor of the camera, was displayed in the centre of the screen. 
Any joystick movement resulted in a corresponding displacement of the background relative 
to the cross-hair. During the experiment, the target (a small diamond, 0,4°) would move 
across the background. Subjects were instructed to control the joystick with their right hand 
thumb and index finger and to keep the visor aimed at the target as well as possible. The 
distance between target and visor during tracking was continuously measured in order to 
indicate performance. 

To simulate the disturbing forces of wind gusts up in the air, the camera platform and the 
target were disturbed by a disturbance function using different starting points within this 
function for both elements. This function caused the platform/camera as well as the target to 
change direction randomly at a rate of 30Hz while they moved at a constant speed of 5.30 
or 2.65cm/s over the screen, respectively. The unfiltered direction changes were limited to a 
maximum of 10° or 20°, depending on the motor taskload condition. The direction changes 
were filtered such that the frequency of direction changes could not exceed 0.625 or 1.25Hz, 
depending on the motor load condition. 

For the dual task, eight small rectangles were placed in a circle at a distance of 4cm around 
the visor. These were always displayed to keep the number of items on the screen constant. 
As explained below, the rectangles were used in manipulating the amount of visual taskload. 


2.4 Independent variables 

In the experiment six factors were varied, two between subjects (platform stabilization and 
CHI), which resulted in the four tracking modes, and four within subjects (update 
frequency, motor load, visual load and replication). 

Automated platform stabilization 

The platform/camera could be disturbed as described above. In the automated stabilization 
conditions this disturbance was completely removed such that the platform remained stable. 
Consequently, in these conditions camera motions were completely generated by subject- 
initiated forces on the joystick. 
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The platfoim/camera could be controlled by either a passive joystick or an acnve joystick 
providing CHI by its motions. The CHI involved information concerning either P atform 
Lability or the system-initiated compensatory actions aimed at nulling the piatform 
instability The amplitude of the active stick movements constantly took up one third of the 
total stick range because platform speed was also constant. The largest possible steering 
input subjects had to give, never exceeded two third of the stick range so that presence of 

CHI did not limit target control. . 

The aim of the information transferred by the stick movements depended on the stick 
condition. While controlling an instable platform, CHI informed the operator on how to 
correct instabilities himself. The position of the stick indicated the direction of platform 
disturbance which could be corrected by steering in the opposite direction When the 
platform was stabilized, CHI informed the operator on the correcting actions of the system. 
Stick position indicated what the system was doing to stabilize the platform. This knowledge 
of the system’s actions would help the operator to maintain a complete representation of all 
flight aspects (keeping the man in the loop) thereby enhancing situation awareness. 


Tracking modes 

The combination of platform stabilization and CHI resulted in the following four tracking 
modes. 

Mode 0 was obtained by combining a passive joystick with an instable camera platform. 
Target tracking movements had to be superimposed on the steering input required for 
platform control. 

Mode 1 involved the same passive control stick. The platform, however, was stabilized 
automatically and would not move unless the stick was moved. 

Mode 2 was similar to mode 0 except for the fact that subjects used an active joystick to 
control the platform. The active joystick provided the subjects with CHI by its tilt. By 
counteracting on the computer-generated joystick movements the instability of the platform 
had to be compensated for. On top of these actions, tracking movements had to be super¬ 
imposed. (In this condition CHI was comparable to the information used by Korteling & 
Van Gent, 1994.) 

Mode 3 combined an active joystick with automatic stabilization. The function of the active 
stick was to give CHI on the compensatory actions made by the computer. Tracking 
movements thus had to be superimposed on the computer-generated stick movements. (CHI 
in this mode was comparable with the haptic information previously used by Korteling & 
Van der Borg, 1994, 1997.) In fact, this condition involved a pursuit target tracking task 
with system-initiated joystick motions that may be expected to produce interference with the 
tracking task. 


Update frequency 

Based on theoretical and practical criteria, update frequency of the monitor image was either 
2, 3, 5 or 30Hz. With update frequencies lower than 5Hz the perception of movement of the 
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target gradually deteriorates until it disappears completely at 2Hz and down. It will cause 
target motion to become unpredictable to the operator (this also depends on the magnitude of 
the target movements). With an update frequency of 30Hz, on the other hand, complete 
image fusion is provided. Therefore, it is regarded as desirable (Massimino & Sheridan, 
1994). At the same time, some UAY systems currently used in military settings are not yet 
able to reach update frequencies above 5Hz (Van Breda & Passenier, 1993). 

Motor load 

The amount of motor taskload was manipulated by varying the range of direction changes 
generated by the random disturbance signal. This counted for the disturbance of the camera 
platform as well as for the target. Direction changes by this signal were limited to a 
maximum of 10° (low load) or 20° (high load) to the left or right and were generated 
randomly at 30Hz.load condition. The direction changes were filtered such that the fre¬ 
quency of direction changes were limited to 0.625Hz in the low-load condition or 1.25Hz in 
the high-load condition. 

Visual load 

The amount of visual taskload was varied by using an extra visual attention task with 
minimal motor demands (Fig. 1). The task was also designed such that it could not generate 
structural visual interference, i.e., by provoking eye movements from the target. This task 
forced subjects to spread their visual attention from the visor in the centre of the screen to 
an area of about 6° in diameter around the visor. Along the edge of this imaginary circle, 
eight small blocks (0.7° diagonal) were positioned. The circle rotated slowly (1.25%) to 
prevent fading of the blocks as a consequence of too strong eye-fixations. 

During a high visual taskload trial, two of the blocks now and then disappeared from view 
for a time of 100ms. Subjects had to judge the smallest area between the disappearing blocks 
and respond only when two blocks were in between (this number of remaining blocks could 
be 0, 1, 2, or 3). This was the case in twelve out of twenty-four stimuli per trial. Responses 
had to be made by pushing a button with the left (i.e., non-dominant) index-finger. If no 
response was initiated within 1.5s, a tone was presented until the button was pressed. In case 
of a false alarm, two short beeps followed the response. Subjects had to keep the number of 
errors and late reactions below 5 %. In order to keep motor demands as low as possible this 
task just required subjects to react accurately. Hence, apart from the time limit, reaction 
time was not relevant. 

Replication 

After the presentation of all 16 within-subjects conditions, in the first 16 trials, all these 
trials were presented once more and, per subject, in the same order to check possible effects 
due to learning or fatigue. 





Fig. 1 The eight blocks around the visor, used for the extra visual attention 
task, displayed on the background consisting of randomly located + signs. 

2.5 Dependent variables and data analysis 

In order to measure tracking performance, Root Mean Squared (RMS) tracking error was 
computed for each trial. Error was defined as the deviation of the centre of the tracking 
symbol on the monitor screen from the centre of the target in pixels on the screen. This 
deviation was sampled by 10Hz. As the first four seconds of each trial were not used, RMS 
was based on a total number of 116sxl0Hz=1160 deviation-scores per trial. On these RMS 
values analyses of variance (ANOVA) were performed. A 2(platform-stabilization) X 
2(CHI) between groups design was used with four repeated measure factors: image update 
rate (4 levels) x motor load (2) x visual load (2) x replication (2). The number of errors 
(incorrect responses or misses) in the dual-task conditions was far below the 5% margin. 
These data will therefore not be further discussed. 
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2.6 Procedure 

The experiment plus training lasted one morning or afternoon (3 hours) during which two 
subjects were tested. At any time, only one subject was performing the task while the other 
subject rested in the adjacent room. 

Practice session 

Before running practice trials, a briefing, involving the reading of a brief instruction, was 
done as to the nature of the experiment and the tasks to be performed (see Appendix B). 
Subsequently, the subjects practised together for about 40 minutes (8 trials of each 120s for 
each subject). In a pilot experiment this was found to be sufficient to achieve stable 
performance for each stick-mode. During the practice session the experimentator answered 
possible questions related to the task. 

Experimental trials 

In the experiment, 32 trials were presented to the subjects, i.e., image update rate (4 levels), 
visual load (2), motor load (2), replication (2). These were divided in four eight-trial blocks. 
For each subject, all trials were performed in the same stick mode (one out of the four 
modes) throughout the experiment. Trial duration was 120s. Between trials subjects paused 
for 20s. After completing each block of trials, the subjects traded places. The order of 
conditions was balanced between subjects. 


3 RESULTS 

3.1 Automated stabilization and CHI 

A main effect of automation showed that tracking performance with an automatically 
stabilized platform was superior relative to an instable platform [F(l,52)=79.279, 
p< < .01]. The passive stick clearly benefitted from automated platform-stabilization. For 
the active stick, this stabilization benefit just failed to reach significance (Fig. 2). 

In addition, a significant main effect for CHI was found indicating that subjects performed 
better in the active joystick modes involving CHI [F(l,52)=5.738, />< .05]. Although 
subjects performed better with CHI, this effect turned out to be limited to the conditions 
without automated platform-stabilization. When the platform was stabilized automatically, no 
significant differences were found between the active and passive stick according to a post- 
hoc analysis (Tukey HSD test)! Stabilization and CHI were thus not additive. Fig. 2 shows 
this interaction [F(l,52)=37.473, p<< .01], Only without stabilization, CHI provided 
better tracking performance. 
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Fig. 2 Tracking error as affected by CHI and platform stabilization. 


3.2 Other independent variables 

The data also showed a significant main effect of image update rate [F(3,156) = 355.95, 
p< <.01]. Tracking performance improved with increasing update frequency of the camera 
image. In Fig. 3, RMS error is displayed for each of the four different image update rates. 
There was a significant motor-load effect indicating an increment in RMS error with 
increasing target motion [F(l,52)=562.526, /?<<.01]. When subjects had to perform the 
extra visual attention task, tracking performance also deteriorated significantly [F(l,52) = 
18.259, p< <0.01], 

No effect of replication was found, showing that performance did not increase (training 
effect) or decrease (fatigue effect) during the experiment. 



Fig. 3 Mean RMS tracking error as affected by update frequency. 
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3.3 Interactions 

Neither visual taskload nor update frequency or tracking difficulty interacted with CHI. 
There was, however, a significant interaction effect of automation x update rate [F(3,156) 
=25.983, p< <.01]. Although the effect remained significant, there was a decrease in the 
benefit of automation with increasing update frequency (Fig. 4a). Automatic stabilization 
also interacted significantly with motor load [F(l,52)=132.967, p<< .01], Tracking 
performance was degraded most by increasing target motion when the platform was 
unstabilized (Fig. 4b). 




Fig. 4 Mean tracking performance with a stabilized and a non-stabilized 
platform as affected by update frequency (a) and motor taskload (b). 



Fig. 5 Mean tracking performance with various update frequencies as affected 
by visual taskload (a) and motor taskload (b). 
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Furthermore, update rate interacted with visual load as manipulated by the visual attention 
task [F(3,156)=9.615, p< <.01]. The difference in RMS tracking error for conditions of 
absence or presence of the visual attention task disappeared at a frequency of 30Hz. Below 
this value, it was significant and of constant magnitude (Fig. 5a). A similar effect was 
observed for the interaction of update frequency x motor load. At 30Hz the (significant) 
difference between the two levels of motor load was smallest (Fig. 5b). 


4 DISCUSSION 

In everyday life people are hardly aware of their dependence on haptic information. It is not 
until other sensory information is degraded or cannot be sampled that touch is fully 
appreciated (e.g. finding a keyhole in the dark or when reaching under one’s seat while 
driving a car). In remote control settings, the operator usually lacks haptic information while 
visual information is restricted and distorted as a consequence of a limited viewing angle and 
low image update frequency and/or resolution. At present, there is still too little knowledge 
regarding the possibilities and limitations of haptic information and the effects of interacting 
factors in order to formulate design principles for the increasing number of remote control 
tasks. Therefore, research on the possibilities and limitations of presentation of haptic 
information in teleoperation should be conducted in order to contribute to this knowledge. 
This simulator study was conducted to provide more insight in these complicating factors in 
platform control, in particular in haptic information provided by control devices. 

In the present experiment haptic information was provided through joystick movements. The 
operator controlled a moving platform-with-camera to track a moving target. The platform 
was considered instable as a consequence of various unpredictable forces such as wind gusts. 
Continuous haptic information (CHI) consisted of stick movements that informed the 
operator on the direction of these forces. In one mode, these forces provided information 
concerning the joystick forces that had to be applied in order to compensate for platform 
disturbances. In this case, platform stabilization was acquired just by counteracting as well 
as possible the system-initiated joystick motions. In one other mode, CHI did not provide 
such information because the platform was already stabilized. This condition was incorpo¬ 
rated in order to test the negative effects of the system-initiated joystick motions. However, 
in practical task settings, this kind of CHI can be beneficial as providing additional informa¬ 
tion (situation awareness) concerning system automation. Automation may degrade situation 
awareness and therefore is considered a significant factor underlying human error and 
accidents (Endsley, 1995; Wickens, 1992). In the present experiment automation consisted 
of (perfect) system ’corrections’ for the external forces disturbing the platform on which the 
camera was mounted, e.g., gyroscopic stabilization. This was called automatic stabilization. 
For the present experiment, it was not expected that this haptic information for increasing 
situation awareness would enhance performance because it was not relevant for the tracking 
task. Actually, this condition involved a pursuit target tracking task entailing system-initiated 
joystick motions that may be expected to interfere with the tracking task. The question here 
involved the degree to which this continuous haptic interference would hamper tracking 
performance as such. If the results would indicate minimal negative effects of joystick 
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motion with automatic platform stabilization, the idea of enhancing situation awareness by 
CHI in semi-automated control tasks would be a promising opportunity. 

Compared to tracking performance in the passive joystick conditions, in which no automatic 
stabilization or CHI was provided, the data showed that subjects performed substantially 
better with both kinds of support. The beneficial effect of automation was most pronounced, 
followed by the combined effects of automation and CHI, CHI only, and finally no support 
at all. It is clear from these results that both haptic information and (of course) automation 
are of help. The effects, however, clearly are not additive as the combined effects of CHI 
and stabilization were not larger than the separate effects. 

To explain the results a closer look at the task dynamics will be helpful. The task at hand 
involved in fact both a pursuit and a compensatory component (tracking the target and 
correcting platform instability, respectively). When the system takes care of stabilization, the 
operator is only responsible for the pursuit tracking component. Therefore performance 
improves substantially with automatic stabilization. The fact that addition of CHI in this 
condition led to a small (insignificant) decrease in tracking performance can be explained on 
the basis of a point made by Kelley (1968). He stressed that continuous active signals disturb 
the operator’s internal model of the steering dynamics and thereby will interfere with the 
operation of a mechanical device. However, because exclusive CHI led to clear performance 
improvement in the non-stabilized condition, it can be concluded that the interfering effects 
of CHI on the pursuit component apparently are smaller than the positive effects of provid¬ 
ing information that can be used for compensating platform disturbances. Therefore, the 
present data indicate that the interference by stick motion, as described by Kelley (1968), 
was relatively small and may be cancelled when positive effects of the active signals (e.g. 
aimed at enhancing situation awareness) are sufficiently large. On the basis of a more 
tentative study, this already was envisioned by Korteling and Van der Borg (1994, 1997). 
Hence, the addition of CHI in case of automatic stabilization of platform disturbances may 
enhance situation awareness, whereas it slightly decreases tracking performance per se. This 
trade-off has to be taken into consideration for each specific man-machine system that uses 
automatic stabilization. For example, an operator of an unmanned aircraft may benefit from 
feeling that the airframe is turning while his outside image remains earth-fixed. 

A pronounced main effect of update frequency was found. The effect of an increase in 
update rate (from 2Hz to 5Hz or from 3Hz to 30Hz) was of the same magnitude as the 
effect of CHI whereas an increase (from 2Hz to 30Hz) equalled the effect of automatic 
stabilization. 

Negative performance effects of extra visual- or motor taskload and positive effects of 
automatic stabilization decreased with increasing update frequency. For the extra visual 
attention task, this interaction can be explained by the fact that with low update rates, the 
task becomes more demanding such that less spare-capacity is left for extra tasks, in this 
case the visual-spatial attention task, or task-complications that depend on the same visuo- 
spatial-manual functions or resources (i.e., Wickens, 1987, 1992). The decreasing effects of 
disturbance motion and automatic stabilization with update frequency can be explained by 
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the fact that low frequencies reduce the amount of visual motion information, which will be 
more detrimental as the amount of motion to be compensated increases. 

The effect of automation was found to be highly advantageous to tracking performance as a 
consequence of eliminating the need for compensatory control by the operator. In other 
words: having to compensate disturbances while tracking a moving target makes the tracking 
task more difficult. As the effect of an increase in platform disturbances and target movabil- 
ity affected both pursuit and compensatory tracking performance, it was observed not 
surprisingly, that increased disturbances were less detrimental to tracking performance in 
conditions of automated platform stabilization. Because the increase in compensatory 
tracking effort was dealt with by the system, the only component suffering was pursuit 
tracking. For the extra visual attention task no such interaction with automation was found, 
that is, dual task performance could not benefit from automation as it was not affected by 
differences in compensatory tracking demands. 

Effects of all independent factors, except CHI, were reduced with increasing update 
frequency. It seems then that the effects of CHI and update frequency are additive. In 
addition, CHI helped independently of motor load, and visual load. This means that CHI 
helps not only under adversive task conditions (as far as the three manipulated here), but 
also under many task conditions in which these factors are more optimal. 

Major conclusion of the present study is that CHI at a control device helps when it involves 
information that can be used directly for vehicle control. In addition, when it provides 
information that is less relevant for control, it only marginally degrades control perfor¬ 
mance. When automation partially takes over control, or parts of the control process are 
allocated to a co-operator, the camera operator is partly separated from the control process 
and may loose situation awareness. This essential aspect of teleoperation may be 
compensated for by providing additional information by the control device that does not 
significantly interfere with the primary control task. In that case the operator will keep 
complete control over the system while being able to cope with potential orientation 
problems. 

One note has to be made in connection to the present simulator study. As a consequence of a 
deliberate choice for a two dimensional presentation a number of (complicating) factors was 
left out of consideration. Therefore it might be that this conclusion is not fully applicable to 
a 3D situation. The results of a study by Korteling and Van der Borg (1994, 1997) permit 
for good prospects. Their conclusion that semi-automation in combination with an active 
interface lead to improved tracking performance compared to a non-aided platform was 
based on experimental results from a 3D tracking task. 

CHI has been shown to be useful under conditions of low image quality and both high visual 
and motor taskload. Therefore it offers a promising opportunity in teleoperation. In sofar as 
high taskload is concerned, partial task automation may be a good option to reduce process¬ 
ing demands. CHI may in this case be used to maintain a sufficient level of situation 


awareness. 
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APPENDIX A Technical specifications of the TNO-TM teleoperation simulator 


Display monitor 


Manufacturer: Mitsubishi, type Diamond Pro 20; 

Principle: CRT video colour monitor, multisync; 

Video input: standard video RGB-S; 

line frequency up to 72 kHz; 
raster frequency up to 100 Hz. 


Display computer 


Manufacturer: 

Principle: 

Resolution: 

Colour: 

Video output: 


Silicon Graphics, type IRIS 4D; 

3D-images of graphical processor, colour; 

1280x1024 pixels at 15 Hz; 

max 1024 colours; 

standard video RGB-S 

line frequency up to 72 kHz. 


Joystick 


Manufacturer: 

Principle: 

Handling force: 
Degrees of freedom: 
Spring characteristic: 
Zero: 

Adjustability: 


Measurement Systems Inc., type 462L miniature; 
force sensor controls servomechanism of joystick; 
maximum 10 N at 25mm stick height; 
two horizontal, one rotation (optional); 
programmable; 

programmable (between -30 and 30°); 
from —30 to 30°. 


Computer systems for supervision en position calculation 


Manufacturer: 

Principle: 

Processors: 

Data registration: 


standard IBM compatible PC; 

network of processors for supervision and position calculation; 
MSDOS system software version 5, clock frequency at least 12 MHz; 
on-line registration real-time sampling, maximum 30 Hz. 
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APPENDIX B Instruction for subjects (in Dutch) 


Instructie trackingexperiment 

Je bevindt je straks in een controlekamer met een monitor, joystick en een aantal knoppen. 
Op de monitor zie je het camerabeeld van een camera die hoog in de lucht onder een 
bewegend platform hangt. De camera is loodrecht naar beneden gericht op een gebied 
waarin een voertuigje rijdt. Het is de bedoeling dat je het voertuig zo precies mogelijk in het 
midden van het camerabeeld houdt. Dit doe je door met de joystick het platform van koers 
te laten veranderen zodat de camera recht boven het voertuigje blijft. 

Als er twee minuten voorbij zijn wordt de missie beeindigd en start de proefleider de 
volgende missie. 

Op de monitor zie je verder nog acht blokjes staan die tezamen een kring vormen. Tijdens 
sommige missies kunnen twee van deze blokjes af en toe korte tijd oplichten wanneer je de 
camera aan het besturen bent. 

Als je dit ziet gebeuren tel je snel het aantal blokjes tussen de twee oplichtende blokjes 
(probeer dit te doen zonder je ogen op de blokjes te richten). Zitten er twee blokjes tussen, 
dan druk je binnen anderhalve seconde op de groene knop. Als je te laat bent hoor je een 
vervelende pieptoon die net zolang duurt tot je alsnog op de groene knop drukt. Zit er maar 
een of zitten er drie blokjes tussen reageer je niet! Druk je toch op de knop, dan hoor je een 
korte dubbele pieptoon. 

Elke pieptoon (of dubbele pieptoon) telt als een fout. Probeer absoluut geen fouten te 
maken!!!! 

Je krijgt zo dadelijk eerst even de gelegenheid om te oefenen. Mocht je vragen hebben kun 
je die altijd aan de proefleider stellen. 
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The present study was conducted to gain insight in the effects of different forms of continuous haptic 
information (CHI) to operator performance of an unmanned platform equipped with a camera-monitor system. In 
a simulator experiment, subjects had to track a moving target keeping a simulated disturbed platform as well 
as possible straight above a moving target. While performing this tracking task, haptic information was 
provided concerning translations of the platform (and thereby of camera). 

Two independent control variables were manipulated between-subjects (four control conditions). These factors 
were: presence or absence of CHI and of (automated) stabilization of the platform. Each of the four groups 
consisted of 14 subjects. Three other factors were varied within-subjects, i.e., image degradation by four 
different image update rates, motor taskload through the amount of target motion, and visual taskload by the 
presence or absence of an extra visual attention-task. During the experiment subjects were seated in front 
of a monitor in a closed room. This screen was used for visual information presentation. The supplementary 
haptic information was generated by movements of a so-called active joystick which at the same time was used 
for steering the camera by "force control". Analysis of the tracking errors shows that CHI as well as 
automated platform stabilization substantially reduced tracking error. These effects were not additive as 
the combined effects of CHI and stabilization were not larger than the separate effects. Whereas the effect 
of stabilization diminished (but remained significant) with increasing update rate, the effects of CHI and 
update rate were additive. With increasing motor or visual taskload, as a consequence of an extra visual 
task or increased target motion, the stabilization effect increased. The magnitude of the CHI effect 
appeared independent of both forms of taskload. 

Major conclusion: CHI at the control device improves tracking performance when it involves information that 
can be used directly for vehicle control; when it provides information that is less relevant for control, 
tracking performance is not substantially degraded. 
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